A risk comparison of ordinary least squares vs ridge regression
نویسندگان
چکیده
We compare the risk of ridge regression to a simple variant of ordinary least squares, in which one simply projects the data onto a finite dimensional subspace (as specified by a principal component analysis) and then performs an ordinary (un-regularized) least squares regression in this subspace. This note shows that the risk of this ordinary least squares method (PCA-OLS) is within a constant factor (namely 4) of the risk of ridge regression (RR).
منابع مشابه
Concentration Reversals in Ridge Regression
Ridge regression is often the method of choice in ill–conditioned systems. A canonical form identifies regions in the parameter space where Ordinary Least Squares (OLS) is problematic. A curious but unrecognized property of ridge solutions emerges: Under spherical errors with or without moments, the relative concentrations of the canonical estimators reverse as the ridge scalar evolves, the est...
متن کاملRidge Regression Estimator: Combining Unbiased and Ordinary Ridge Regression Methods of Estimation
Statistical literature has several methods for coping with multicollinearity. This paper introduces a new shrinkage estimator, called modified unbiased ridge (MUR). This estimator is obtained from unbiased ridge regression (URR) in the same way that ordinary ridge regression (ORR) is obtained from ordinary least squares (OLS). Properties of MUR are derived. Results on its matrix mean squared er...
متن کاملPerformance of Ridge Regression Estimator Methods on Small Sample Size by Varying Correlation Coefficients: a Simulation Study
When independent variables have high linear correlation in a multiple linear regression model, we can have wrong analysis. It happens if we do the multiple linear regression analysis based on common Ordinary Least Squares (OLS) method. In this situation, we are suggested to use ridge regression estimator. We conduct some simulation study to compare the performance of ridge regression estimator ...
متن کاملAn Analysis of Random Design Linear Regression
The random design setting for linear regression concerns estimators based on a random sample of covariate/response pairs. This work gives explicit bounds on the prediction error for the ordinary least squares estimator and the ridge regression estimator under mild assumptions on the covariate/response distributions. In particular, this work provides sharp results on the “out-of-sample” predicti...
متن کاملTracking Mse Efficiencies in Ridge Regression
Ridge regression is often favored in the analysis of ill-conditioned systems. A canonical form identifies regions in the parameter space where Ordinary Least Squares (OLS) is problematic. The objectives are two-fold: To reexamine the view that ill-conditioning necessarily degrades essentials of OLS; and to reassess ranges of the ridge parameter k where ridge is efficient in mean squared error (...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Journal of Machine Learning Research
دوره 14 شماره
صفحات -
تاریخ انتشار 2013